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Abstract 

Incredible  loiter  times  coupled  with  the  ability  to  make  extremely  detailed 
collections  at  significant  stand-off  distances  with  a  relatively  expendable  platform  has 
made  demand  for,  and  diversity  of  RPA  operations  grow  at  voracious  rates.  Conversely, 
financial  resources  are  becoming  increasingly  constrained.  As  such  innovators  are 
looking  to  maximize  the  effectiveness  of  existing  personnel  and  assets  by  considering 
concepts  such  as  simultaneous  Multiple  Aircraft  Control  (MAC)  by  a  single  aircrew. 

Research  has  identified  procedural  inefficiencies  in  current  operations  as  well  as 
substantial  impediments  to  MAC  implementation  including  dynamic  task  saturation  and 
communication  challenges.  An  identified  inefficiency  afflicting  both  current  operations 
and  the  feasibility  of  MAC  is  the  time  required  to  transfer  operational  situation  awareness 
at  shift  change  -  dubbed  “change-over”.  The  present  research  employed  synergistic 
application  of  Cognitive  Task  Analyses,  Situation  Awareness  Oriented  Design  and 
simulation  to  inform  the  development  of  a  highly  efficient  user-centered  process  for  the 
Mission  Intelligence  Coordinator  -  the  RPA  aircrew’s  situation  awareness  linchpin. 
Discrete-event  simulations  were  performed  on  existing  and  proposed  protocols.  These 
analyses  indicate  that  the  proposed  protocol  could  require  as  little  as  one-third  the  time 
required  by  the  current  method.  It  is  proposed  that  such  an  improvement  could 
significantly  increase  current  RPA  mission-readiness  as  well  as  diminish  a  known 
obstacle  to  MAC  implementation. 
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STREAMLINING  THE  CHANGE-OVER  PROTOCOL  FOR  THE  RPA  MISSION 
INTELLIGENCE  COORDINATOR  BY  WAY  OF  SITUATION  AWARENESS 
ORIENTED  DESIGN  AND  DISCRETE  EVENT  SIMULATION 


I.  Introduction 


Background 

Military,  government  and  commercial  industries  across  the  globe  have  only 
recently  begun  to  realize  the  potential  benefits  of  medium  and  high-altitude  Remotely 
Piloted  Aircraft  (RPA)  technologies.  Consequently  the  demand  for  RPA  operations, 
especially  within  the  US  Air  Force,  has  grown  at  insatiable  rates.  In  a  2010  federal 
hearing  on  unmanned  systems  and  the  future  of  war,  US  House  of  Representatives 
Subcommittee  on  National  Security  and  Foreign  Affairs  Chainnan  John  F.  Tierney 
quantified  current  trends,  “As  the  United  States  is  engaged  in  two  wars  abroad, 
unmanned  systems,  particularly  unmanned  aerial  vehicles,  have  become  a  centerpiece  of 
that  war  effort.  In  recent  years,  the  Department  of  Defense’s  inventory  has  rapidly  grown 
in  size,  from  167  in  2002  to  over  seven  thousand  today.  Last  year,  for  the  first  time,  the 
US  Air  Force  trained  more  unmanned  pilots  than  traditional  fighter  pilots.”  (Tierney, 
comments  made  23  March  2010).  Furthermore,  the  US  Air  Force  Unmanned  Aerial 
System  Flight  Plan  2009-2047  (pp  15)  stated,  “UAS  have  experienced  explosive  growth 
in  recent  history,  providing  one  of  the  most  “in  demand”  capabilities  the  USAF  presents 
to  the  Joint  Force.  The  attributes  of  persistence,  efficiency,  flexibility  of  mission, 
infonnation  collection  and  attack  capability  have  repeatedly  proven  to  be  force 
multipliers  across  the  spectrum  of  global  Joint  military  operations.” 

Clearly,  Department  of  Defense  (DoD)  leaders  have  come  to  recognize  the  power 
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and  economy  of  RPA  technologies,  and  see  them  as  strategic  solutions  to  the  omnipresent 
requirement  to  find  new  ways  to  do  far  more  with  far  less.  Battlefield  commanders  relish 
the  thought  of  persistent  and  detailed  real-time  knowledge  of  activities  on  the  battlefield 
while  strategically  deploying  limited  ground  resources.  Today’s  front-line  warrior  yearns 
for  the  substantial  tactical  advantages  of  timely  and  effective  close  air  support  (CAS) 
from  their  RPA  escorts.  At  all  levels,  RPA  technologies  have  become  imperative  to 
modem  warfare,  and  there  is  little  doubt  that  the  robotic  extension  of  the  human 
warfighter  has  permanently  changed  the  way  battles  are  planned,  waged  and  won. 

Problem  Statement 

The  immense  power  of  these  technologies  has  led  to  an  unrelenting  appetite  for 
RPAs  to  do  even  more  -  more  sorties,  more  flight  hours,  more  data  collection,  more 
coverage.  The  reality,  however,  is  to  do  more,  substantial  resources  must  be  dedicated 
and  expended.  In  today’s  fiscal  environment,  this  simply  is  not  feasible  and  innovators 
have  been  charged  to  find  RPA  force-multiplying  efficiencies  to  assist  in  bridging  current 
and  foreseeable  resource/demand  gaps  with  ever  decreasing  resources. 

To  that  end,  research  is  ongoing  to  maximize  the  effective  utilization  of  existing 
RPA  assets,  up  to  and  including  the  concept  of  a  single  aircrew  controlling  multiple 
aircraft  simultaneously,  referred  to  as  Multiple  Aircraft  Control  (MAC).  A  common 
challenge  facing  both  MAC  and  current  operations  is  the  amount  of  time  existing 
protocols  require  to  efficiently  transfer  situation  awareness  (SA)  at  shift  change  -  dubbed 
“change-over”.  Analyses  have  concluded  that  change-over  activities  can  consume  up  to 
10%  of  a  pilot’s  mission  time  in  single-aircraft  control  (Schneider  &  McGrogan,  2011). 
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In  a  MAC  scenario,  if  a  pilot  accepts  control  of  three  or  four  aircraft,  the  subsequent 
transfer  time  and  effort  can  rapidly  balloon  to  consume  an  unreasonable  percentage  of  the 
pilot’s  effective  mission  time.  Similar  challenges  and  results  are  anticipated  for  the  rest  of 
the  aircrew  to  include  the  sensor  operator  and  mission  intelligence  coordinator  (MICs). 
Such  a  reduction  in  effectiveness  runs  counter  to  the  intended  objectives  of  MAC 
implementation. 

Research  Objectives 

The  objective  of  the  present  research  is  to  use  disciplined  and  pedigreed  methods 
to  reduce  the  duration  of  time  required  to  accomplish  the  transfer  of  operational  SA  from 
a  losing  crew  to  a  gaining  crew  at  change-over  in  single  aircraft  control.  Doing  so  could 
increase  the  mission  capability  of  current  RPA  forces,  as  well  as  validate  the  use  of  such 
methods  to  inform  the  subsequent  design  of  a  MAC  change-over  protocol. 

Furthennore,  RPA  industry-wide,  literature  does  not  yet  exist  on  change-over 
processes  and  specific  design  recommendations  despite  medium  and  high-altitude  RPAs 
being  used  in  numerous  civil  and  military  applications.  Similarly,  a  literary  gap  exists 
with  respect  to  the  operational  information  requirements  of  the  RPA  MIC.  While  such 
analyses  have  been  conducted  on  pilots,  particularly  in  manned  aircraft,  no  such  research 
has  been  published  with  respect  to  the  role  of  the  MIC.  This  research  serves  to  address 
these  gaps. 
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Research  Focus 


The  primary  purpose  of  change-over  is  for  the  on-coming  crew  to  achieve 
sufficient  understanding  of  the  current  state  of  all  operationally  pertinent  information 
prior  to  assuming  full  control  of  the  aircraft.  While  the  duration  required  to  accomplish 
change-over  is  an  issue  affecting  the  entire  aircrew,  the  decision  was  made  to  focus  the 
present  analysis  on  the  role  of  the  MIC.  The  MIC’s  primary  role  is  to  act  as  the  team’s 
communication  focal  point  -  integrating,  filtering  and  passing  infonnation  between  the 
aircrew  and  the  numerous  external  parties.  Both  the  pilot  and  sensor  operator  have 
responsibilities  that  often  fully  tax  their  individual  cognitive  resources  -  particularly 
during  dynamic  events  such  as  targeting,  weapons  employment  or  an  in-flight 
emergency.  It  is  during  these  times,  ironically,  that  the  speed,  volume  and  relevance  of 
communication  are  the  most  severe.  Third  parties  including  the  squadron  chain  of 
command,  intelligence  sources,  friendly  ground  and  air  forces,  and  the  customer  must 
exchange  extremely  time-sensitive  infonnation  continuously  and  reliably  with  the  RPA 
crew.  Any  missed  communication  or  delay  of  message  receipt  could  very  easily  mean  the 
difference  between  mission  success  and  catastrophic  failure. 

In  these  situations,  communication  directly  translates  to  situation  awareness 
exchange  -  both  amongst  the  crew  and  the  external  parties.  It  is  for  this  reason  that  the 
MIC,  as  the  aircrew’s  primary  communication,  and  therefore  SA  processor,  was  chosen 
as  the  focus  of  the  analysis. 
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Investigative  Question 

The  present  research  attempts  a  redesign  of  the  RPA  MIC  change-over  protocol 
from  the  current  technology-centered  process  to  a  user-centered  process.  The  redesign  is 
predicated  upon  Situation  Awareness  Oriented  Design  (SAOD)  protocols  and  principles. 
The  basic  investigative  question  is,  how  can  the  current  change-over  process  be 
redesigned  to  facilitate  more  expedient  transfer  of  SA?  It  is  hypothesized  that  converting 
the  design  to  a  user-centered  design  could  lead  to  a  significant  reduction  in  process 
duration,  thereby  increasing  RPA  operational  availability. 

Methodology 

This  analysis  consisted  of  five  primary  steps.  First,  a  cognitive  task  analysis 
(CTA)  of  the  current  MIC  change-over  process  was  completed  to  identify  each  of  the 
individual  root  knowledge  elements,  or  essential  elements  of  information  (EEIs),  which 
must  be  exchanged  at  change-over  for  a  receiving  MIC  to  achieve  sufficiently  operational 
SA.  In  other  words,  the  CTA  effort  identified  each  of  the  specific  data  points  that  a 
gaining  MIC  must  know  to  be  able  to  safely  and  successfully  assume  control  of  a 
mission.  The  second  step  was  the  creation  and  evaluation  of  the  current  process 
architecture  model  within  Rockwell  Automation’s  discrete-event  simulation  software 
Arena.  Within  this  step,  each  of  the  process’s  individual  tasks  were  enumerated  and 
represented.  Duration  estimates  and  distributions  for  each  were  obtained  from  subject 
matter  experts  and  built  into  the  model.  Arena  was  then  used  to  conduct  Monte-Carlo 
simulations  to  gain  insight  into  the  duration  and  variability  of  the  process.  Next,  the 
SAOD  principles  and  heuristics  were  applied  to  the  various  types  of  root  data  identified 


5 


by  the  CTA,  enabling  highly  efficient  and  disciplined  design  of  an  improved  change-over 
construct,  carefully  built  around  the  goal  of  effective  and  efficient  SA  achievement.  The 
fourth  step  was  to  translate  the  newly  designed  change-over  system  definition  into  Arena 
for  evaluation.  Finally,  statistical  analysis  on  the  output  from  the  current  and  proposed 
Arena  model  runs  was  accomplished  with  resultant  conclusions  and  operational 
implications  drawn. 

Assumptions/Limitations 

For  the  current  change-over  process  model,  subject  matter  experts  provided  input 
and  subsequently  validated  distributions  representing  the  time  duration  typical  of  each 
individual  EEI  discussion  during  nonnal  operations.  These  durations  were  individually 
built  into  the  respective  Arena  module  representing  the  appropriate  EEI.  Similarly,  for  the 
proposed  model,  our  subject  matter  experts  assisted  with  the  generation  and  validation  of 
distributions  for  the  expected  duration  to  accurately  glean  the  needed  EEIs  from  each  of 
the  reports,  displays  and  processes  proposed  within  the  new  model.  In  both  the  current 
and  proposed  model,  subject  matter  experts  validated  triangular  distribution  parameters. 
Ideally,  robust  and  quantitative  work  studies  should  be  conducted  with  numerous  real- 
world  users  on  representative  systems  conducting  realistic  scenarios  to  gain  highly 
accurate  time  measurements.  The  limitations  of  this  study  were  such  that  the  level  of 
rigor  and  the  resources  needed  to  conduct  said  work  studies  were  infeasible. 

In  line  with  the  stated  goals  of  the  present  research,  efficiency  was  the  sole 
dependent  variable  in  question.  Thus,  the  analysis  was  limited  to  the  metric  of  process 
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efficiency  (time),  with  the  understood  caveat  that  additional  research  is  needed  to  address 
the  system  effectiveness  portion  of  the  equation. 

Preview 

This  thesis  employs  a  scholarly  fonnat.  As  such,  an  article  produced  to  describe 
and  publish  the  research  is  presented  in  the  following  chapter.  The  article  is  presented  as 
formatted  for  submission  to  the  Human  Factors  and  Ergonomics  Society  Journal  of 
Cognitive  Engineering  and  Decision  Making.  Because  of  the  intent  to  have  the  article 
published  in  a  public  venue,  the  research  was  presented  in  such  a  way  as  to  demonstrate 
its  applicability  to  a  broader  audience  beyond  the  niche  of  the  US  Air  Force.  As  such, 
military  vernacular  and  potentially  sensitive  data  were  removed.  Most  notably,  the  Air 
Force-unique  position  title  of  MIC  was  changed  to  the  more  general  title  of 
Communications  Officer  (CO)  -  though  the  understood  roles  remained  unchanged. 
Similarly,  activities  and  EEIs  that  specifically  pertain  to  military  operations  were  omitted 
or  made  more  general.  For  instance,  “supported  units”  became  “customers”,  “Restricted 
Operating  Zones”  (ROZs)  became  “clearances”,  and  “kill  box/keypad/altitude”  data 
became  simply  “location  data  in  three  dimensions”.  Conversely,  terms  like  targets, 
payloads  and  threats  still  hold  true  in  civilian  applications  and  as  such  were  not  altered. 
For  example  a  real  estate  company  employing  an  RPA  to  photograph  properties  would 
still  have  targeted  locations,  a  payload  of  one  or  several  cameras  and  potential  threats 
including  power  lines  (during  take-off  or  landing  phases)  or  known  areas  of  electro¬ 
magnetic  signal  interference. 
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Though  the  research  has  been  abstracted  to  a  generalized  level,  the  conclusions 
and  resultant  recommendations  as  discussed  in  the  article  are  readily  applicable  across 
both  civilian  and  military  domains. 
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II.  Scholarly  Article 


Formatted  for  submission  to  the  Journal  of  Cognitive  Engineering  and  Decision  Making 

Application  of  Cognitive  Task  Analysis  and  Process  Models  to  Streamline  Operator 

Change-over 

John  Machuca,  Michael  Miller,  John  Colombi,  Randall  Gibb 


Abstract 

Remotely  piloted  aircraft  (RPAs)  provide  highly  effective  capabilities  -  long 
loiter  time,  detailed  sensing  from  great  distance  and  an  expendable  platform.  RPAs 
actively  support  military  operations,  homeland  defense,  firefighting,  search  and  rescue 
operations,  geophysical  surveys,  and  commercial  surveillance.  Research  is  ongoing  to 
maximize  the  utilization  of  existing  assets  in  current  and  future  operations,  up  to  and 
including  the  concept  of  simultaneous  Multiple  Aircraft  Control  (MAC)  by  a  single 
aircrew.  A  common  challenge  facing  both  MAC  and  current  operations  is  the  efficient 
transfer  of  situation  awareness  at  shift  change  -  dubbed  “change-over”.  This  paper  details 
the  integration  of  Cognitive  Task  Analysis,  Situation  Awareness  Oriented  Design,  and 
discrete-event  simulation  to  inform  the  design  of  a  highly  efficient  user-centered  change¬ 
over  protocol.  The  resultant  protocol  was  modeled,  simulated  and  measured,  yielding  a 
mean  duration  one -third  that  of  current  methods;  a  highly  significant  result  for  current 
operations  and  the  feasibility  of  MAC.  The  subsequent  results  can  serve  as  a  baseline 
change-over  protocol. 
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Introduction 


Semi-autonomous  Remotely  Piloted  Aircraft  (RPA)  have  transfonned  the  modern 
world,  making  it  tremendously  more  accessible,  surveillable,  manageable  and  survivable 
for  those  that  employ  them  effectively.  The  most  powerful  currency  in  nearly  any  market 
or  endeavor,  including  warfare,  is  information  -  and  in  tactical-terms  information 
translates  directly  to  situation  awareness  (SA).  The  protracted  RPA  loiter  times  enable 
extensive  and  comprehensive  data  collection,  with  the  end  result  being  situation 
awareness  the  likes  of  which  decision  makers  have  never  before  enjoyed. 

In  the  course  of  the  last  decade,  leaders  within  the  US  Department  of  Defense 
(DoD),  Department  of  Homeland  Security  (DHS),  US  Forest  Service,  and  many  other 
governments  and  commercial  industries  have  come  to  recognize  the  power  and  economy 
that  RPA  technologies  offer  -  and  RPA  prevalence  has  surged  accordingly.  In  a  2010 
federal  hearing  on  unmanned  systems  and  the  future  of  war,  US  House  of  Representatives 
Subcommittee  on  National  Security  and  Foreign  Affairs  Chairman  John  F.  Tierney 
stated,  “As  the  United  States  is  engaged  in  two  wars  abroad,  unmanned  systems, 
particularly  unmanned  aerial  vehicles,  have  become  a  centerpiece  of  that  war  effort.  In 
recent  years,  the  Department  of  Defense’s  inventory  has  rapidly  grown  in  size,  from  167 
in  2002  to  over  seven  thousand  today.  Last  year,  for  the  first  time,  the  U.S.  Air  Force 
trained  more  unmanned  pilots  than  traditional  fighter  pilots.”  (Tierney,  comments  made 
23  March  2010) 

Although  the  US  military,  more  specifically  the  Air  Force,  is  the  best  known 
employer  of  RPA  technologies,  they  are  far  from  the  only.  In  2007,  NASA  operated  a 
modified  Predator  B  to  aid  in  firefighting  efforts  in  southern  California  (GAO-08-5 11, 
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2008).  In  2005,  NASA,  the  National  Oceanographic  and  Atmospheric  Administration  and 
industry  partnered  to  use  RPAs  in  the  study  of  climate,  water  resource  forecasting,  costal 
mapping  and  ecosystem  monitoring  and  management.  The  US  Department  of  Homeland 
Security  Customs  and  Border  Protection  credits  its  RPAs  with  assisting  agents  to 
complete  over  4,000  arrests  and  seizure  of  nearly  20,000  pounds  of  illegal  drugs  between 
September  of  2005  and  March  of  2008.  With  renewed  volcanic  activity  at  Mount  St. 
Helens  in  Washington  in  2004,  the  US  Geological  Survey  and  Forest  Service  used  an 
RPA  to  collect  data,  operating  well  above  the  heat  and  toxic  gases  of  the  volcano. 

Commercial  applications  of  RPAs  are  also  being  actively  exploited  and 
continuously  furthered.  Oil  companies  and  geophysicists  have  used  RPAs  to  more 
quickly  and  efficiently  identify  previously  undiscovered  mineral  deposits  by  seeking  out 
underlying  rock  structures  via  RPA-borne  cesium  magnetometer  (magnetic  field) 
measurements  (source:  http://www.universalwing.com/technology/our-uav).  Commercial 
surveillance  of  pipelines,  livestock,  and  roadways  occurs  daily.  The  real  estate  industry 
has  employed  RPAs  to  collect  and  offer  detailed  photography  of  plots  of  land.  RPA 
technologies  have  become  pervasive  across  numerous  industries  and  operations,  and  the 
upward  trends  show  no  signs  of  slowing. 

In  today’s  fiscal  environment,  ample  demand  is  not  often  accompanied  by  ample 
resources.  And  as  resources  become  more  and  more  constrained,  RPA  employers  seek  to 
utilize  their  existing  assets  as  effectively  as  possible. 

To  find  ways  to  do  so,  one  must  first  understand  that  medium  and  high-altitude 
RPAs  operate  within  a  complex  system-of-systems  architecture.  The  complete  “RPA 
system”  typically  consists  of  one  or  more  air  vehicles,  ground  control  stations  for  both 
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primary  mission  control  and  take  off/landing,  a  communications  suite  (including 
intercom,  chat,  radios,  phones,  a  satellite  link,  etc),  support  equipment,  and  command, 
operations  (aircrew)  and  maintenance  crews  (US  Air  Force  MQ-1B  Predator  Fact  Sheet, 
2010)  which  are  often  distributed  globally.  The  aircrew  for  these  medium  and  high- 
altitude  RPAs  with  sensor  (or  weapons)  payloads  typically  consist  of  a  two  or  three- 
person  team  depending  on  mission  workload;  a  vehicle  pilot,  a  sensor  operator  and  in 
more  dynamic  applications  a  communications  officer  (CO).  As  one  would  expect,  the 
pilot  is  primarily  responsible  for  the  control  and  operation  of  the  physical  aircraft.  The 
sensor  operator’s  responsibilities  lie  with  the  operation  of  the  various  sensor  payloads 
that  are  installed  on  the  aircraft.  These  duties,  though  interdependent,  can  task  saturate 
the  respective  crewmember  rapidly,  particularly  during  a  dynamic  event  such  as 
targeting,  active  data  collection  or  an  in-flight  emergency.  It  is  during  these  times, 
ironically,  that  the  speed,  volume  and  relevance  of  communication  are  at  the  most  severe. 
During  complex  operations  where  extensive  amounts  of  communication  takes  place 
amongst  an  ever-changing  number  of  internal  and  external  parties,  the  CO  (or  in  Air 
Force  tenns,  Mission  Intelligence  Coordinator)  acts  as  the  aircrew’s  communication  and 
global  SA  focal  point.  The  MIC  synthesizes,  filters  and  passes  infonnation  from  and  to 
the  dozens  of  entities  involved,  thereby  allowing  the  pilot  and  sensor  operator  to  more 
diligently  focus  on  their  highly  dynamic  and  cognitively  taxing  responsibilities. 
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Figure  1.  Aircrew  Interaction  Diagram 


Problem  Statement 

Personnel  resources,  particularly  RPA  pilots,  sensor  operators  and 
communications  officers,  often  prove  to  be  a  nontrivial  limitation  in  today’s  resource 
constrained  environment.  Thus,  research  is  ongoing  to  maximize  the  utilization  of 
existing  personnel  and  hardware  assets,  up  to  and  including  the  operational  concept  of 
simultaneous  Multiple  Aircraft  Control  (MAC)  by  a  single  aircrew.  Research  has 
identified  procedural  inefficiencies  in  current  operations  as  well  as  substantial 
impediments  to  MAC  including  dynamic  task  saturation  and  communication  challenges 
(Schneider  &  McGrogan,  2011).  An  identified  challenge  common  to  both  current 
operations  and  MAC  is  the  efficient  transfer  of  situation  awareness  at  shift  change  - 
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dubbed  “change-over”.  With  mission  durations  exceeding  20  hours  for  medium  and  high- 
altitude  RPAs,  change-over  is  an  event  that  often  occurs  more  than  once  during  any  given 
sortie  and  the  effectiveness,  completeness  and  duration  of  the  SA  exchange  can  have 
immense  impact  on  the  success  or  failure  of  an  on-going  mission. 

Past  analyses  have  concluded  that  the  duration  of  change-over  activities  using 
current  protocols  can  consume  up  to  10%  of  an  RPA  pilot’s  mission  time  in  single¬ 
aircraft  control  (Schneider  &  McGrogan,  2011).  In  a  MAC  scenario,  if  a  pilot  accepts 
control  of  three  or  four  aircraft,  the  subsequent  transfer  time  and  effort  can  rapidly 
balloon  to  consume  an  unreasonable  percentage  of  the  pilot’s  effective  mission  time. 

Such  a  reduction  in  effective  mission  time  runs  counter  to  the  intended  objectives  of 
MAC  implementation. 

Similar  challenges  are  anticipated  for  the  rest  of  the  aircrew.  As  the 
communication  and  SA  integration  and  dissemination  focal  point  there  are  a  substantial 
and  unpredictable  number  of  parties  that  the  CO  will  interact  with  during  the  span  of  a 
given  mission.  Couple  that  with  the  highly  dynamic  nature  of  their  mission,  and  it 
becomes  clear  that  the  effective  and  efficient  transfer  of  the  requisite  SA  data  points  to  an 
on-coming  CO  at  change-over  is  a  significant  challenge. 

Research  Objectives 

The  objective  of  the  present  research  is  to  use  pedigreed,  user-centered  methods 
to  reduce  the  duration  of  time  required  to  accomplish  the  transfer  of  sufficient  SA  from  a 
losing  aircrew  to  a  gaining  aircrew.  Doing  so  could  serve  to  increase  the  effective 
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mission  availability  of  existing  RPA  forces,  as  well  as  inform  the  use  of  such  methods  in 
the  creation  of  future  protocols,  including  for  MAC. 

Furthennore,  RPA  industry-wide,  literature  does  not  yet  exist  on  change-over 
processes  and  specific  design  recommendations  despite  medium  and  high-altitude  RPAs 
being  used  in  numerous  civil  and  military  applications.  Similarly,  a  literary  gap  exists 
with  respect  to  the  operational  information  requirements  of  the  RPA  CO.  While  such 
analyses  have  been  conducted  on  pilots,  particularly  in  manned  aircraft,  no  such  research 
has  been  published  with  respect  to  the  role  of  the  CO.  This  research  serves  to  address 
these  gaps. 

Investigative  Question 

The  present  research  attempts  a  redesign  of  the  RPA  CO  change-over  protocol 
from  the  current  technology-centered  process  to  a  user-centered  process.  The  redesign  is 
predicated  upon  Situation  Awareness  Oriented  Design  (SAOD)  protocols  and  principles. 
The  basic  investigative  question  is,  how  can  the  current  change-over  process  be 
redesigned  to  facilitate  more  expedient  transfer  of  SA?  It  is  hypothesized  that  converting 
the  design  to  a  user-centered  design  could  lead  to  a  significant  reduction  in  process 
duration,  thereby  increasing  RPA  operational  availability. 

Background 

Situation  Awareness 

“Few  issues  in  aviation  psychology,  or  in  the  larger  arena  of  engineering 
psychology,  generate  the  controversy  that  is  commonly  associated  with  the  topics  of 
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mental  workload  and  situation  awareness.”  (Vidulich,  2003,  pp  115)  Even  beyond  the 
verbiage  of  the  definition,  whether  the  tenn  SA  should  refer  to  a  discrete  cognitive 
product  which  results  from  a  process  or  to  the  continuously  updating  process  itself 
(Flach,  1995)  is  a  point  of  substantial  contention. 

With  that  in  mind,  the  intent  of  this  analysis  is  to  design  a  brief  process  used  to 
establish  a  sufficient  “snap-shot”  understanding  of  the  operational  situation,  from  scratch, 
for  an  in-coming  aircrew  member.  The  “snap-shot”  pertains  to  a  discrete  moment  in  time. 
Thus,  the  system  definition  itself  is  discrete  by  nature  and  is  not  intended  to  continually 
update  SA  over  time,  but  rather  generate  a  relatively  terse  and  basic  comprehension.  To 
that  end,  the  term  SA  in  this  paper  shall  refer  to  the  cognitive  product  which  results  from 
the  processes  being  analyzed,  as  opposed  to  the  processes  themselves. 

On  a  more  practical  level,  at  the  heart  of  any  definition  of  SA  is  the  comparison 
between  actual  system  status  and  the  operator’s  perception  and  understanding  of  system 
status  (Woods,  1988).  As  this  gap  widens,  SA  deteriorates.  As  it  closes,  SA  becomes 
more  complete  and  actionable.  Endsley’s  definition  of  situation  awareness  has  stood  the 
test  of  time  and  follows  as  “the  perception  of  the  elements  in  the  environment  within  a 
volume  of  time  and  space,  the  comprehension  of  their  meaning  and  the  projection  of  their 
status  in  the  near  future”  (Endsley,  1988,  pp  792).  She  goes  on  to  describe  three  discrete 
levels  of  SA  -  perception,  comprehension,  and  projection.  The  first  level,  perception, 
considers  the  capture  of  critical  factors  or  data  points  describing  the  environment. 
Integrating  those  factors  into  a  cohesive  understanding  of  the  situation  constitutes  the 
second  level  of  SA.  The  third  and  highest  level  of  SA  is  using  the  infonnation  from  the 
previous  two  levels  to  project  into  the  future  how  the  situation  might  change,  thereby 
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enabling  development  of  contingency  plans  in  case  of  such  changes.  Each  level  of  SA  is 
progressively  more  difficult  to  attain  than  the  previous. 

Maintaining  sound  and  complete  SA  is  a  task  of  paramount  importance  to  any 
vehicle  operator.  Without  sufficient  understanding  of  one’s  physical  surroundings, 
system  location  and  orientation  (in  all  three  axes),  system  state  of  health  infonnation,  the 
location  and  profile  of  any  possible  threats  or  allies,  and  so  on,  an  operator  has  little  to  no 
hope  of  successfully  completing  an  assigned  task,  or  perhaps  even  surviving  the  mission. 
Because  medium  and  high-altitude  aircraft  operate  several  miles  above  terra  firma,  to  say 
maintaining  sound  SA  is  very  important  to  an  aircrew  is  a  dramatic  understatement.  Any 
decision  made  during  a  lapse  of  SA,  even  a  momentary  one,  can  quickly  result  in 
catastrophic  consequences.  A  review  by  Hartel,  Smith  and  Prince  (1991)  found  SA  to  be 
the  leading  causal  factor  of  military  aviation  mishaps.  Of  major  air  carrier  accidents 
involving  human  error,  88%  could  be  attributed  to  SA  failures  per  a  report  conducted  by 
Endsley  in  1994.  These  dangers  are  dramatically  asseverated  in  RPA  egocentric 
teleoperation  (operator  is  physically  removed  from  the  vehicle  and  controls  the  craft 
without  having  line-of-sight).  Physical  disconnection  from  the  system  creates  a  host  of 
additional  SA  issues  for  RPA  crew  to  include  lack  of  visual,  auditory  and  tactile  sensory 
cues,  time  lags  between  command  issuance,  vehicle  response  and  feedback,  “blind  spots” 
in  system  status  data,  signal  interruption,  degradation  or  loss,  sensor  malfunctions,  and 
system  interface  challenges  to  include  the  ongoing  mental  task  of  creating  a  fused  sight 
picture  from  disparate  data  sources.  Riley  &  Endsley  (2005)  go  on  to  expand  this  list 
adding  “out-of-the-loop  syndrome”,  mode  awareness  problems,  and  vigilance  decrements 
to  the  SA  challenges  faced  by  RPA  crews. 
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Situation  Awareness  Oriented  Design 

Endsley  et  al  (2003b)  present  Situation  Awareness  Oriented  Design  (SAOD)  as  a 
method  to  improve  human  decision-making  and  operational  perfonnance  by  enhancing 
the  ability  to  maintain  situation  awareness.  "The  key  is  in  understanding  that  true 
situation  awareness  only  exists  in  the  mind  of  the  human  operator.  Therefore  presenting  a 
ton  of  data  will  do  no  good  unless  it  is  successfully  transmitted,  absorbed  and  assimilated 
in  a  timely  manner  by  the  human  to  form  situation  awareness."  (Endsley  et  al,  2003a,  pp 
1)  Using  SAOD,  system  designs  are  constructed  with  deliberate  and  pedigreed  provisions 
to  support  efficient  and  effective  high-level  SA  achievement  in  order  to  enhance  an 
operator’s  decision  making  abilities  and  consequent  task  performance.  The  process 
begins  by  assessing  the  SA  requirements  involved  in  the  task,  typically  by  way  of  a  CTA. 
Those  resultant  SA  requirements  are  then  translated  into  system  design  requirements. 

Specifically,  the  process  is  comprised  of  three  main  facets:  SA  requirements 
analysis,  SA-oriented  design  principles,  and  SA  measurement. 


SA  Requirements 
Analysis 

h 

SA-Orlented 
Design  Principles 

SA  Measurement 

Figure  2.  Situation  Awareness  Oriented  Design  Process 

The  SA  requirements  analysis  portion  of  the  process  addresses  the  task  of 
determining  what  it  is  the  operator  needs  to  know  at  any  given  point  of  a  task  to  have 
sufficient  and  operable  situation  awareness.  SA  requirements  can  be  further  defined  as 
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the  requisite  dynamic  information  data  points  (as  opposed  to  static  knowledge  such  as 
rules,  policies,  etc)  that  an  operator  would  need  in  order  to  achieve  all  goals  and  sub¬ 
goals  of  their  task(s),  often  referred  to  as  essential  elements  of  infonnation  (EEIs).  It  is 
important  to  note  that  this  analysis  is  not  strictly  limited  to  identifying  the  individual 
dynamic  data  points,  but  may  also  need  to  include  careful  consideration  for  how  that 
information  is  presented  and  integrated  when  making  a  decision.  The  tool  of  choice  to 
accomplish  this  phase  of  the  analysis  is  typically  an  appropriate  fonn  of  Cognitive  Task 
Analysis  (CTA),  which  is  detailed  in  the  next  section. 

The  crux  of  SAOD  is  taking  the  products  from  the  SA  requirements  analysis 
portion  of  the  process  and  then  applying  the  fifty  SA-Oriented  Design  principles  (Endsley 
et  al,  2003b)  which  were  developed  based  upon  a  theoretical  model  of  the  methods  used 
in  acquiring  and  maintaining  SA  in  dynamic  complex  systems.  The  guidelines  are 
"focused  on  a  model  of  human  cognition  involving  dynamic  switching  between  goal- 
driven  and  data-driven  processing  and  feature  support  for  limited  operator  resources" 
(Endsley  et  al,  2003a,  pp  2).  Topics  addressed  in  the  heuristics  include,  but  are  not 
limited  to,  supporting  SA  in  multi-warfighter  and  distributed  team  environments,  design 
of  alarms,  dealing  with  the  complexity  of  systems,  design  of  advanced  automation 
concepts,  and  so  on.  A  sampling  of  the  individual  principles  includes: 

Principle  1  -  Goal  oriented  information  displays  should  be  provided  and 
organized  so  that  the  infonnation  needed  for  a  particular  goal  is  co-located  and 
directly  answers  the  major  decisions  associated  with  the  goal. 
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Principle  3  -  Whenever  possible  provide  direct  presentation  of  higher-level 
SA  needs  (comprehension  and  projection  activities)  as  opposed  to  simply 
providing  low  order  data  for  the  operators  to  integrate  and  interpret  themselves. 

Principle  23  -  Provide  consistency  and  standardization  on  controls  across 
different  displays  and  systems. 

Principle  35  -  Use  automation  for  assistance  in  carrying  out  routine 
actions  rather  than  higher  level  cognitive  tasks. 

Principle  47  -  Provide  flexibility  to  support  shared  SA  across  functions. 
For  a  full  list  of  the  SAOD  principles,  see  Endsley,  Bolte  and  Jones  (2003b). 

The  generic  nature  of  the  principles  and  heuristics  ensures  the  universal 
applicability  of  SAOD  to  any  system  design  where  SA  plays  a  role.  Clearly,  every 
principle  will  not  be  applicable  to  every  system  design,  but  typically  a  multitude  of  the 
principles  will.  The  flexibility  and  pedigree  of  the  SAOD  principles  has  been  explicitly 
demonstrated  by  successful  application  in  the  fields  of  remote  maintenance  operations, 
medicine,  manufacturing,  and  military  command  and  control. 

The  SA  Design  measurement  phase  of  the  SAOD  process  is  where  direct  or 
indirect  measurement  of  SA  is  attempted  to  gauge  success  of  the  effort  and  the  potential 
for  further  design  improvements.  Recommended  SA  measurement  methods  include  the 
indirect  methods  of  verbal  protocols,  communication  analyses,  psychophysiological 
metrics,  behavioral  measures,  and  performance  outcome  measures.  Direct  measurement 
methods  include  self-ratings,  Situational  Awareness  Rating  Technique  (SART)  (see 
Taylor,  1990),  observer  ratings,  post-test  questionnaires,  on-line  real-time  questionnaires 
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and  the  Situation  Awareness  Global  Assessment  Technique  (SAGAT)  (see  Endsley, 
1988). 

These  methods  attempt  to  measure  the  effectiveness  of  a  SA  system  design.  In  a 
departure  from  typical  SAOD  protocols,  the  present  research  takes  for  granted  the 
effectiveness  of  the  system  due  to  the  proven  and  validated  pedigree  of  the  EEIs  and 
focuses  on  the  efficiency  of  the  design  in  a  specific  attempt  to  minimize  the  expected 
process  duration.  To  measure  this  dependent  variable,  statistical  analysis  was 
accomplished  on  the  output  of  discrete-event  simulations  conducted  within  Rockwell 
Automation’s  Arena  Simulation  Software. 

Cognitive  Task  Analysis 

Borne  of  the  applied  psychology  field,  cognitive  task  analysis  is  a  broadly  applied 
and  often  customized  kit  of  tools  and  techniques  used  for  the  identification  and 
description  of  the  knowledge  and  strategies  needed  for  task  accomplishment.  Per 
Shraagen  et  al  (2000,  pp  3),  “Cognitive  task  analysis  is  the  extension  of  traditional  task 
analysis  techniques  to  yield  infonnation  about  the  knowledge,  thought  processes,  and 
goal  structures  that  underlie  observable  task  performance.”  While  CTA  has  numerous 
manifestations,  structures  and  levels  of  fonnality,  typically  the  process  can  be  described 
as  five  steps  which  are  briefly  detailed  in  the  following  paragraphs. 

At  the  outset  of  a  CTA,  preliminary  data  collection  should  be  accomplished.  The 
intention  of  this  first  step  is  for  the  analyst  to  become  familiarized  with  the  domain  and 
process  being  studied  and  includes  the  identification  of  key  cognitive  tasks,  with 
particular  attention  paid  to  those  tasks  that  are  difficult,  frequent  or  highly  critical 
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cognitive  tasks  within  the  job.  To  achieve  these  ends,  the  tools  and  techniques  at  the 
analyst’s  disposal  include  literature  review,  observations  and  unstructured  interviews.  At 
the  conclusion  of  this  effort  it  is  generally  beneficial  to  visually  depict  any  results,  often 
in  the  form  of  knowledge  or  concept  maps  describing  the  specific  infonnation 
requirements,  EEIs,  and  any  relevant  relationships  between  tasks  and  sub  tasks.  The 
creation  of  such  knowledge  representations  comprises  the  second  step  of  the  procedure. 

With  education  of  the  domain,  the  process  and  its  tasks  attained  and  represented, 
the  analyst  can  then  begin  the  third  step  of  a  more  intelligent  and  focused  investigation  of 
the  EEIs  by  implementing  applied  knowledge  elicitation  methods  which  include 
interviews,  both  unstructured  and  structured,  verbal  protocol  walkthroughs  and  the 
Critical  Decision  Method  (see  Klein  et  al  for  additional  information  on  the  Critical 
Decision  Method).  Of  these,  structured  and  unstructured  interviews  are  most  commonly 
selected  as  they  require  little  training  and  are  relatively  easy  to  employ  and  customize  to 
the  desired  level  of  investigative  fonnality. 

Next  the  analyst  must  take  the  infonnation  that  has  been  collected  thus  far  and 
synthesize  it  into  meaningful  conclusions.  As  part  of  this  fourth  step,  the  resultant  data 
should  be  refined,  packaged  and  presented  in  an  intuitive  way  to  facilitate  taking  the 
synthesized  infonnation  and  conclusions  and  providing  it  to  subject  matter  experts  to 
have  the  data  verified  for  its  intended  purpose.  With  the  data  and  resultant  conclusions 
validated,  the  yield  of  actionable  infonnation  to  infonn  intelligent  system  design  will 
have  been  delivered. 

The  fifth  and  final  step  of  the  CTA  is  to  take  the  achieved  results  and  to  translate 
them  into  meaningful  and  infonnative  models  which  should  “reveal  the  underlying  skills, 
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mental  models,  and  problem-solving  strategies  used  by  experts  when  performing  highly 
complex  tasks”  (Clark  et  al,  2008,  pp  582).  These  models  can  then  be  used  to  inform 
curriculum,  training  procedures  and  system  design. 

Change-over 

Lacking  a  well-defined  industry  standard  change-over  protocol,  this  study  used 
US  Air  Force  protocols  as  the  baseline.  The  US  Air  Force  has  one  of  the  most  robust, 
mature  and  heavily  utilized  change-over  procedures  in  the  industry.  Furthermore,  the  Air 
Force  is  actively  grappling  with  the  economic  constraints  previously  discussed  as  the 
impetus  of  the  research  and  is  actively  seeking  out  methods  to  maintain  or  even  increase 
operational  capabilities  while  simultaneously  reducing  resources  including  manpower.  A 
general  change-over  protocol,  as  documented  by  visits  and  interviews  conducted  with 
RPA  operators,  was  selected  to  serve  as  the  baseline  for  this  analysis.  No  single  person  or 
specific  organization’s  protocol  was  selected.  Rather,  to  account  for  variation  as  well  as 
to  reduce  the  sensitivity  of  the  infonnation,  the  process  was  generalized  to  capture  only 
the  ubiquitous  characteristics  that  are  relevant  across  all  medium  and  high-altitude  RPA 
operations,  both  federal  and  civilian. 

Some  US  Air  Force  RPAs  have  sustainable  loiter  times  greater  than  20  hours 
(Chappelle  et  al,  2010).  Therefore,  operational  RPA  squadrons  typically  operate  multiple 
shifts  to  provide  the  necessary  aircrew  coverage.  As  such,  change-overs,  being  defined  as 
the  procedural  transfer  of  situation  awareness  and  RPA  control  from  a  losing  crew  to  a 
gaining  crew,  are  a  common  occurrence  that  often  takes  place  multiple  times  during  the 
course  of  any  given  RPA  tasking. 
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In  terms  of  duration  and  therefore  operational  impact,  analysis  has  shown  that 
change-overs  can  account  for  8-20%  of  total  mission  time  in  single  aircraft  control,  and 
have  been  projected  to  take  up  to  40%  of  total  mission  time  if  a  MAC  of  four  were 
employed  (Schneider  &  McGrogan,  2011).  Furthermore,  several  military  RPA  mishaps 
have  been  attributed  to  failures  either  during  or  as  the  result  of  a  change-over  (Tvaryanas 
et  al,  2005).  These  analyses  aptly  highlight  the  challenge  that  efficient  and  effective 
change-over  presents  to  modern  RPA  operations. 

It  is  important  to  note  that  during  these  change-over  periods  the  attention  of  the 
out-going  crew  is  split  amongst  conducting  change-over  discussions  with  the  in-coming 
crew  and  continued  support  of  the  ongoing  taskings.  As  such,  all  reasonable  efforts  are 
made  to  avoid  conducting  change-over  during  any  dynamic  situation  such  as  an  in-flight 
emergency,  targeting,  signal  or  optical  collections,  weapons  employment,  and  so  on. 

The  baseline  RPA  change-over  process  in  this  report  is  detailed  as  three  distinct 
phases:  the  in-coming  aircrews  receive  a  mass  pre-mission  brief  from  the  mission  support 
cell,  each  individual  oncoming  crew  member  receives  their  individualized  change-over 
briefs  from  their  respective  losing  crew  member  within  the  ground  control  station,  and 
finally  the  members  of  a  gaining  crew  complete  an  internal  crew  brief  to  establish  plans 
of  action. 

Phase  1  -  Gaining  crew  receives  pre-mission  brief  from  mission  support  cell 

Approximately  30  minutes  prior  to  the  scheduled  change-over  time,  the  mission 
support  cell  provides  the  gaining  crew  with  a  pre-mission  brief  to  educate  them  on 
pertinent  mission  data  (mission  assignment,  intelligence  reports,  weather  reports,  etc), 
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rules  of  engagement  and  recent  mission  developments.  This  step  is  intended  to  provide 
the  on-coming  crew  with  macro-level  SA  prior  to  progressing  to  subsequent  steps. 

Generalized,  this  phase  provides  context  and  top-level  strategy  data.  For  example 
in  a  commercial  topographic  cartography  mission,  the  crew  would  be  informed  of  the 
target  area  to  be  mapped,  the  desired  image  types  and  fidelities  and  so  on. 

Phase  2  -  Change-over  brief 

This  step  is  where  the  bulk  of  SA  infonnation  is  exchanged.  Typically,  the  losing 
CO  utilizes  electronic  or  hardcopy  checklists,  handwritten  flight  logs/notes  and  current 
system  displays  to  provide  a  verbal  brief  of  all  relevant  mission  and  vehicle  data.  Topics 
discussed  include  weather,  airspace  data,  datalink  infonnation,  emergency  mission 
information,  and  target  information.  The  authority  to  declare  this  step  complete  resides 
with  the  gaining  CO.  This  prevents  the  losing  CO  from  departing  the  area  until  the 
gaining  has  self-declared  satisfactory  operational  situation  awareness. 

Generalizing,  this  step  focuses  on  tactical-level  infonnation.  Recalling  the 
example  cartography  mission,  this  would  include  cunent  equipment  (payload)  status  and 
configuration  infonnation,  information  on  pertinent  relationships  with  external  parties, 
system  maintenance  issues  or  anomalies,  detailed  target  data  and  so  on. 

Phase  3  -  Gaining  crew  conducts  crew  brief 

With  the  new  crew  in  their  seats  and  in  control  of  the  aircraft,  an  internal  crew 
brief  is  conducted.  During  this  time,  the  crew  discusses  key  items  of  interest  identified  in 
previous  steps  and  protocols  of  internal  task  allocation  termed  “contracts”.  These 
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contracts  are  typically  the  pilot’s  directions  to  the  other  crew  members  on  how  he  or  she 
would  like  things  to  operate,  as  the  pilot  is  typically  the  team  lead.  For  instance,  in  case 
of  an  in-flight  emergency,  the  pilot  may  wish  to  allocate  responsibility  for  all 
communication  with  external  parties  to  the  CO  to  enable  the  rest  of  the  crew  to  focus 
entirely  on  the  dynamic  task  at  hand.  Another  common  contract  put  in  place  is  for  all 
textual  communication  to  occur  in  a  designated  chat  room  visible  to  all,  as  opposed  to 
pennitting  “whispers”,  which  are  chat  rooms  established  from  one  party  directly  to 
another  so  as  to  have  information  passed  out  of  view  from  uninvolved  parties.  A  pilot  can 
establish  crew  contracts  as  he  sees  fit  and  does  so  to  create  clear  ground  rules  with 
respect  to  how  the  crew  shall  conduct  business.  These  contracts  in  and  of  themselves 
have  dramatic  impact  on  the  crew’s  team  SA  and  often  are  directly  pertinent  to  the  key 
elements  of  information  each  crew  member  needs  to  keep  track  of  to  maintain 
satisfactory  operational  shared  SA.  With  contracts  in  place  and  a  baseline  of  team  SA 
established  the  change-over  process  is  declared  complete. 

In  a  general  application,  this  final  step  is  comparable  to  the  designated  crew  lead 
(typically  the  pilot  or  a  shift  manager)  providing  final  direction  with  respect  to  phase  1 
(strategy  and  context)  and  phase  2  (system  and  target  status)  information. 

Methodology 

Overview 

To  achieve  the  stated  research  objectives  the  present  research  employed  Cognitive 
Task  Analysis  (CTA)  to  identify  a  comprehensive  list  of  the  root  information  COs  require 
to  achieve  operational  SA  during  change-over,  which  were  then  leveraged  against  the 
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Situation  Awareness  Oriented  Design  (SAOD)  principles  to  facilitate  pedigreed 
improvement  to  the  system’s  architecture  keeping  the  overall  user-centered  goal  of  timely 
and  effective  SA  transfer  in  mind.  In  line  with  the  focus  of  the  research,  but  in  contrast  to 
typical  SAOD  measurement  methods,  the  resulting  framework  was  modeled  and 
measured  via  discrete-event  simulation.  Finally,  the  simulation  results  were  compared  to 
paired  data  from  the  current  design  and  conclusions  were  drawn. 

Step  1.  Cognitive  Task  Analysis  of  current  change-over  process 

Cognitive  Task  Analysis  was  employed  to  dissect  and  catalog  the  knowledge 
elements  that  must  be  exchanged  at  change-over  in  order  for  the  receiving  CO  to  be  able 
to  achieve  sufficiently  operational  SA. 

The  preliminary  data  collection  phase  of  the  CTA  was  accomplished  via  robust 
literature  review  on  the  topic  of  RPA  team  operations  followed  by  unstructured 
interviews  and  operational  observations  of  active  US  Air  Force  COs.  Knowledge 
representations  were  created  to  describe  the  tasks,  subtasks  and  information  requirements 
observed.  These  representations  facilitated  the  third  CTA  step  of  conducting  fonnalized 
in-depth  interviews  and  operational  observations  of  COs  to  form  a  solid  understanding  of 
the  change-over  process.  In  this  step  the  root  essential  elements  of  information,  EEIs,  or 
the  lowest-level  bits  of  infonnation  needed  for  a  CO  to  achieve  operational  situation 
awareness,  were  identified  and  explored.  These  EEIs  were  then  traced  in  two  fashions  - 
chronologically,  in  that  they  were  each  allocated  to  their  appropriate  time-ordered  step(s) 
of  the  current  process,  and  by  user-centered  goals  in  that  each  EEI  was  allocated  to  the 
higher  level  task  it  supports,  which  in  turn  were  rolled-up  to  the  overarching  user 
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operational  goals  that  those  tasks  serve.  The  top-level  CO  goals  identified  were  to 
facilitate  aviation  (keeping  the  RPA  airborne),  navigation,  communication,  finding  the 
target,  fixing  on  the  target  (locking  sensors  on  to  the  target),  tracking  the  target,  targeting 
the  target  with  the  appropriate  sensors,  engaging  the  target  (making  a  collection  or  in  the 
military  realm  employing  ordinance),  and  assessing  the  results.  By  tracing  EEIs  to 
chronological  steps  as  well  as  user  tasks  and  goals,  any  missing,  redundant,  sequentially 
“misplaced”  or  non-value  added  EEIs  are  highlighted  infonning  subsequent  analyses. 

In  the  final  step  of  the  CTA,  the  data  and  results  of  the  process  were  packaged  and 
presented  to  the  RPA  subject  matter  experts  for  validation  and  verification.  The  subject 
matter  experts  were  active  communications  officers  (Air  Force  MICs)  and  were  equally 
split  amongst  officer  and  enlisted  members  with  flight  qualifications  equally  divided 
amongst  MQ-1  Predator  and  MQ-9  Reaper  aircraft.  US  Air  Force  active  duty,  reservists, 
and  Air  National  Guard  components  were  each  represented. 

Step  2.  Creation  of  current  process  architecture  model 

With  the  CTA  completed,  a  robust  and  detailed  understanding  of  the  current 
change-over  process  was  achieved  and  a  model  built.  Rockwell  Automation’s  Arena 
Software,  a  graphical  discrete-event  simulation  tool,  was  used  to  build  and  analyze  the 
models.  Both  the  current  and  proposed  models  were  constructed  with  a  chronological 
flow  of  the  EEIs  as  addressed  during  the  change-over  process.  In  other  words,  the  out¬ 
going  CO  would  begin  the  process  at  the  “Start”  module  in  the  model,  and  progress  along 
a  singular  path  addressing  each  EEI  in  order,  concluding  the  process  upon  arrival  to  the 
“Finish”  module. 
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Figure  3.  Current  Change-over  Process  Arena  Model 
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Depicted  within  the  model  the  segregated  phases  of  the  process  -  mass  brief, 
change-over  brief  and  crew  brief,  can  be  seen.  Also,  individual  EEIs  that  are  repeated 
(addressed  at  least  once  prior  in  the  model)  are  outlined  in  an  unshaded  box  to  clearly 
depict  the  amount  of  redundancy  in  the  current  process.  With  that  said,  undoubtedly 
redundancy  is  often  both  beneficial  and  intentional.  Redundancy  is  often  used  as  a 
method  to  reinforce  key  infonnation,  provide  opportunity  for  updates  to  dynamic 
information,  or  confirm  that  multiple  parties  have  the  same  sight  picture  throughout  an 
exchange.  With  the  identification  of  substantial  EEI  redundancy  in  the  current  process,  a 
request  was  put  to  the  subject  matter  experts  to  point  out  any  EEIs  for  which  redundancy 
could  be  justifiably  beneficial.  Only  the  EEIs  pertaining  to  weather  were  identified  as 
advisable  to  revisit  with  time.  The  operational  implications  of  weather  are  dramatic  and 
weather  systems  in  many  parts  of  the  world  can  change  rapidly.  The  duration  of  the 
current  process  creates  the  need  to  revisit  this  topic.  The  repetition  of  all  other  EEIs 
identified  as  redundant  were  considered  to  be  non-value  added  tasks. 

Step  3.  Application  of  Situation  Awareness  Oriented  Design  principles 

With  the  CTA  accomplished  and  root  EEIs  identified,  the  SA  requirements  were 
translated  into  system  design  requirements.  The  building  blocks  of  the  new  process  were 
now  in  hand.  To  facilitate  disciplined  and  pedigreed  design  of  the  new  process  using 
these  building  blocks,  the  principles  and  protocols  of  SAOD  were  brought  to  bear.  While 
nearly  all  of  the  50  principles  informed  the  design  in  some  way,  the  following  is  a  brief 
discussion  of  the  five  SAOD  principles  that  offered  the  greatest  relevance  and  impact  to 
this  analysis. 
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Principle  1  -  Organize  infonnation  around  goals.  This  principle  asserts  that  the 
operator’s  major  goals  constitute  the  framework  around  which  all  information 
requirements  and  activities  should  be  centered.  Often,  this  is  not  the  case  and  infonnation 
is  presented  based  entirely  upon  the  sensor  or  system  creating  the  data  (e.g.,  fuel  level,  oil 
temperature).  “Information  should  be  organized  so  that  the  information  needed  for  a 
particular  goal  is  co-located  and  directly  answer  the  major  decisions  associated  with  the 
goal.”  (Endsley  et  al,  2003b,  pp  83)  To  this  end,  the  goals  identified  during  the  CTA 
process  became  the  focal  points  to  which  the  various  EEIs  were  allocated. 

Principle  2  -  Present  level  2  infonnation  directly  -  support  comprehension.  “As 
attention  and  working  memory  are  limited,  the  degree  to  which  displays  provide 
information  that  is  processed  and  integrated  in  terms  of  level  2  SA  requirements  will 
positively  impact  SA.”  (Endsley  et  al,  2003b,  pp  83)  For  example,  it  is  much  more 
intuitive  to  directly  display  the  difference  between  actual  airspeed  and  required  airspeed 
given  the  cunent  climb  angle,  as  opposed  to  simply  stating  the  cunent  airspeed  and 
expecting  the  operator  to  “do  the  math.”  This  principle  was  particularly  applicable  while 
detennining  the  method  of  presentation  to  be  used  to  convey  certain  EEIs.  For  example,  it 
was  detennined  to  be  more  useful  to  display  details  such  as  the  aircraft’s  required  stand¬ 
off  orbit  from  a  given  target  visually  in  a  geographical  display  as  opposed  to  via  verbal 
protocols. 

Principle  3  -  Provide  assistance  for  level  3  SA  projections.  Similar  to  the  previous 
principle,  this  heuristic  reduces  the  level  of  cognitive  effort  required  on  the  part  of  the 
operator  to  go  from  a  lower-level  understanding  of  current  states  to  an  integrated  mental 
model  predicting  future  states  and  trends.  Trend  graphs  and  the  like  facilitate  an 
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operator’s  ability  to  forecast  what  may  be  down  the  road.  Offering  time-phased  data  in  a 
clear  manner,  such  as  a  graphical  display,  has  much  greater  utility  than  a  simple  snap  shot 
capturing  only  current  status,  as  you  would  likely  receive  from  a  terse  verbal  statement 
on  the  matter. 

Principle  39  -  Make  modes  and  system  states  salient.  System  states  and  modes  are 
pivotal  pieces  of  infonnation  that  can  be  the  difference  between  a  given  data  point  being 
within  normal  bounds  or  being  a  sign  of  imminent  danger.  Mode  confusion  is  a  common 
problem  when  directly  operating  nearly  any  vehicle  and  the  issue  is  tremendously 
asseverated  in  egocentric  teleoperation  of  an  aircraft.  Thus,  diligent  effort  was  made 
throughout  this  analysis  to  provide  for  presenting  system  states  and  modes  as  saliently  as 
possible.  The  primary  instantiation  of  this  principle  in  the  improved  process  has  been 
dubbed  the  RPA  Data  Display.  This  display  directly  illustrates  key  EEIs  pertaining  to  the 
state  of  the  air  vehicle  and  the  ground  control  station  -  aircraft  starting  and  current 
payload,  for  example.  To  convey  these  EEIs,  the  envisioned  display  depicts  a  silhouette 
of  the  RPA  with  the  starting  payload  shown  on  each  of  the  respective  pylons.  If  ordinance 
were  dropped  during  the  mission,  the  ordinance’s  image  on  the  respective  pylon  would 
go  from  a  stark  solid  filled  image  to  a  shaded  silhouette.  Such  an  image  would  make  it  so 
that  in  a  brief  glance  one  could  quickly  understand  what  payload  the  aircraft  took  off 
with,  currently  has,  and  even  some  information  on  potential  weight/balance  implications 
that  may  come  with  a  change  in  payload  symmetry  across  the  mission. 

Principle  45  -  Build  a  common  picture  to  support  team  operations.  Obviously,  the 
MIC  is  only  one  part  of  the  aircrew  team.  And  the  MIC’s  change-over  EEIs  are  not  the 
same  as  those  for  the  pilot  or  sensor  operator.  Thus,  when  it  comes  to  SA  needs,  one  size 
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does  not  fit  all.  So  the  change-over  process  as  designed  here  is  not  intended  to  be 
precisely  replicated  and  applied  to  the  pilot  and  sensor  operator’s  change-over  processes 
(nor  as  absolute  truth  across  all  industries  -  some  degree  of  customization  may  be 
necessary).  However,  the  overlap  of  the  EEIs  that  are  common  amongst  the  team 
members  (and  industries)  is  nontrivial,  and  value  can  be  derived  from  developing  systems 
that  facilitate  a  common  sight  picture  by  creating  a  common  data  source  for  at  least  a 
subset  of  the  SA  infonnation  each  crew  member  needs.  In  the  improved  design  this 
principle  is  demonstrated  by  the  Mission  Data  Report,  the  Geospatial  Display  and  the 
RPA  Data  Display.  These  data  sources  contain  information  that  the  entire  team  needs  to 
be  aware  of.  Mission  details  including  target  information,  desired  collection  or 
engagement  affects,  rules  of  engagement,  intelligence  reports,  and  area  threats  will  be 
contained  in  the  Mission  Data  Report  and  all  members  of  the  team  would  count  these 
pieces  of  infonnation  amongst  their  respective  change-over  EEIs.  Similarly,  the  entire 
crew  will  need  at  least  a  basic  geospatial  awareness  of  where  the  aircraft  is,  where  the 
target  is,  the  intended  stand-off  orbit  from  the  target  given  a  particular  detection  concern, 
what  airspace  clearances  do  they  have,  where  the  threats  are,  and  where  each  of  those 
data  points  are  with  respect  to  one  another.  Thus  a  common  display  that  can  be  reviewed 
by  all  members  of  the  aircrew  would  do  well  to  facilitate  a  common  sight  picture  of 
geospatial  SA. 

Step  4.  Creation  of  the  new  process  model 

Analyzing  the  identified  EEIs  in  light  of  the  SAOD  principles,  weaknesses  of  the 
current  change-over  system  are  highlighted  and  methods  to  improve  the  process,  by 
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focusing  the  design  on  the  overarching  goal  of  SA  exchange  and  fonnation,  are  realized. 
Using  the  principles,  especially  those  discussed  above,  it  became  clear  that  an  improved 
CO  change-over  system  should  incorporate  generated  textual  reports  for  relatively  static 
mission  infonnation,  intuitive  visual  displays  of  all  geospatial  infonnation,  and  salient 
system  displays  depicting  dynamic  system  modes  and  states.  The  current  change-over 
protocol  is  technology-centered  in  that  it  has  the  relevant  EEIs  for  these  portions  of 
overall  SA  scattered  throughout  the  process.  This  is  largely  because  the  data  is  grouped 
and  reviewed  according  to  its  respective  technology.  This  results  in  an  unintuitive  and 
nonintegrated  presentation  of  the  data,  inhibiting  level  2  and  3  SA  formation.  SAOD 
principles  point  out  the  value  of  arranging  such  a  process  around  the  user’s  overarching 
goals  and  tasks  so  as  to  expedite  level  2  and  3  SA  achievement.  Rectifying  these  issues 
and  the  elimination  of  non-value  added  tasks  has  yielded  the  following  change-over 
system  definition. 
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Figure  4.  Proposed  Change-over  Process  Arena  Model 
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Not  all  of  the  change-over  EEIs  are  ideally  expressed  by  way  of  a  generated 
report  or  visual  display,  as  found  in  the  mission  data  report,  geo-spacial  display  or  RPA 
system  data  display.  These  EEIs  are  often  nuanced,  difficult  to  quantify  and  even 
subjective  in  nature  at  times.  With  this  sort  of  information  two-way  discussion  is  often 
required  to  ensure  clarity  and  understanding.  The  last  three  potions  of  the  proposed 
process  account  for  these  sorts  of  EEIs.  These  steps  are  intended  to  be  carried  out  face-to- 
face  in  the  ground  control  station  between  the  losing  and  gaining  COs.  These  three  data 
collections  are  the  only  portions  of  the  proposed  design  where  the  losing  CO’s  efforts 
must  be  split  amongst  the  change-over  process,  and  continued  support  of  an  on-going 
sortie.  The  first  three  steps  can  be  accomplished  by  the  gaining  CO  outside  of  the  ground 
control  station  independent  of  the  losing  CO  provided  the  reports  and  displays  are  made 
available. 

Step  5.  Statistical  analysis  of  current  and  proposed  model  output 

In  line  with  the  SAOD  process  described  previously,  the  third  phase  of  the 
process,  SA  measurement,  was  conducted.  To  accomplish  this  step  in  accordance  with 
typical  SAOD  guidelines,  representative  systems  and  interfaces  would  be  built,  realistic 
and  diverse  scenarios  developed  and  experienced  operators  would  conduct  trials  for 
accurate  SA  measurements  to  be  taken.  With  that  said,  the  typical  methods  of  SA 
measurement  mentioned  previously  (psychophysiological  metrics,  SART,  SAGAT)  are 
intended  to  measure  the  effectiveness  (accuracy  and  completeness)  of  SA  generation  and 
maintenance.  Rather,  the  focus  of  the  present  research  is  placed  squarely  on  the 
efficiency  (time  duration  being  the  sole  metric)  of  the  methods. 
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The  method  selected  to  measure  and  analyze  the  efficiency  of  the  processes  was 
Monte  Carlo  discrete-event  simulation.  For  the  current  model,  subject  matter  experts 
provided  input  and  subsequently  validated  distributions  for  the  duration  of  each 
individual  EEI  related  task.  These  durations  were  individually  captured  in  the  respective 
model.  For  the  proposed  model,  our  subject  matter  experts  assisted  with  the  generation  of 
and  subsequently  validated  triangular  distributions  for  the  expected  duration  to  accurately 
collect  the  needed  EEIs  from  each  of  the  reports,  displays  and  processes  in  the  proposed 
model.  Statistical  analyses  of  the  results  were  conducted  on  the  output  from  Arena  for 
each  of  500  replications  for  both  baseline  and  proposed  models  using  synchronized 
random  number  seeds. 

Assumptions  and  limitations 

In  both  the  current  and  proposed  model,  the  subject  matter  expert  validated 
triangular  distribution  duration  estimations  for  the  EEIs  were  at  best,  estimations.  This 
presents  an  admitted  shortcoming  of  the  analysis.  Ideally,  robust  and  quantitative  work 
studies  should  be  conducted  with  numerous  real-world  users  on  representative  systems 
conducting  highly  realistic  scenarios  to  gain  highly  accurate  time  measurements.  The 
limitations  of  this  study  were  such  that  the  level  of  rigor  and  the  resources  needed  to 
conduct  said  work  studies  were  infeasible. 

In  line  with  the  stated  goals  of  the  present  research,  analysis  was  limited  to 
process  efficiency  (time),  with  the  understood  caveat  that  additional  research  is  needed  to 
address  the  system  effectiveness  portion  of  the  equation.  Clearly,  the  effectiveness  as 
well  as  the  efficiency  of  any  system  must  be  considered  to  make  an  accurate  and  fully 
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informed  decision  on  whether  a  design  truly  offers  valuable  improvements,  or  is  even 
acceptable  for  use.  However  in  this  case,  efficiency  was  the  sole  dependent  variable  in 
question. 

Part  of  the  motive  for  this  analysis  was  to  diminish  the  obstacle  that  change-over 
poses  to  MAC  implementation.  However,  this  analysis  was  perfonned  solely  on  single 
aircraft  control  systems.  While  logical  arguments  can  assert  that  lessons  learned  presently 
will  be  able  to  directly  infonn  highly  efficient  MAC  change-over  process  design,  a  robust 
analysis  on  true  MAC  implications  is  warranted. 

Results 

For  the  current  change-over  process  a  mean  duration  time  of  1960  seconds  (32.67 
minutes)  with  a  standard  deviation  of  187  seconds  and  a  95%  confidence  interval  of  ±16 
seconds  was  calculated.  For  the  proposed  process,  a  mean  duration  time  of  639  seconds 
(10.65  minutes)  with  a  standard  deviation  of  78  seconds  and  a  95%  confidence  interval  of 
±7  seconds  were  calculated,  yielding  a  mean  difference  between  the  models  of  1321 
seconds  (22.02  minutes)  with  a  standard  deviation  of  202  seconds.  Conducting  a  paired  t- 
test,  the  mean  reduction  in  process  duration  was  proven  to  be  significantly  greater  than 
zero;  t(499)  =  2.4,  one-tail  p  =  0.009.  A  95%  confidence  interval  about  the  reduction  in 
total  process  duration  comes  out  to  be  (1339,  1303)  seconds. 
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Conclusion 

The  overarching  goal  of  the  effort  was  to  increase  RPA  mission  effectiveness  by 
way  of  providing  crews  with  a  faster  method  to  transfer  SA  at  change-over.  An  additional 
intent  was  to  have  the  resultant  protocol  be  industry-independent  and  to  serve  as  a 
change-over  process  baseline.  To  do  this,  careful  consideration  had  to  be  paid  in 
detennining  the  precise  data  points  that  must  be  exchanged  for  the  recipient  to  have 
sound  and  operationally  actionable  SA  on  the  system  and  its  context.  With  those  data 
identified,  a  sound  understanding  of  pedigreed,  user-centered  principles  proven  to 
facilitate  the  generation  and  maintenance  of  situation  awareness  fostered  dramatic  clarity 
and  insight  into  potential  system  improvements.  The  results  of  the  calculations  performed 
to  measure  the  forecasted  improvements  of  the  proposed  system  were  decisive  and 
unambiguous  -  with  this  study’s  proposed  change-over  protocol  a  67.4%  estimated 
reduction  in  the  time  required  to  accomplish  change-over  could  be  possible  for  AF  assets. 
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Conclusions  such  as  these  are  relevant,  timely,  and  powerful  across  all  RPA  industries, 
but  now  require  operational  validation. 

The  power  of  the  results  of  this  analysis  lie  in  the  potential  to  increase  RPA 
mission  effectiveness  and  availability  in  single  aircraft  control  by  reducing  the  time 
burden  placed  on  current  assets  during  change-over,  as  well  as  serving  to  dissolve  some 
of  the  barrier  that  SA  transfer  poses  to  the  feasibility  of  MAC  implementation.  However, 
it  is  well  worth  noting  that  the  findings  of  is  effort  have  valuable  relevance  beyond  the 
aviation  industry.  For  instance  in  the  medical  field,  doctors  and  nurses  must  conduct 
change-over  processes  to  relinquish  and  assume  responsibility  for  patients.  Additional 
applications  include  industries  such  as  nuclear  operations,  chemical  manufacturers,  and 
chemical  users  with  processes  times  that  span  several  shifts. 

Additionally,  no  CTA-based  research  has  yet  been  published  on  the  RPA  CO  role 
or  on  change-over  protocol  design  despite  their  criticality  to  the  success  of  RPA  missions. 
This  report  addresses  those  gaps. 

Furthennore,  the  synergistic  coupling  of  SAOD  and  discrete-event  simulation  to 
specifically  measure  the  efficiency  of  a  resultant  design  as  opposed  to  the  effectiveness  of 
the  design  is  novel  and  this  work  represents  the  first  known  publicized  demonstration  of 
such  a  tactic. 

Future  Research 

While  the  proposed  system  put  forth  by  this  analysis  asserts  clear  and 
demonstrated  improvements,  further  and  more  rigorous  analyses  are  needed  to  fully  vet 
designs  and  further  optimize  features.  Particular  areas  of  future  research  include  an  in- 
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depth  quantitative  analysis  of  both  the  current  and  proposed  system  design  task  times  to 
calibrate  the  here-in  surmised  duration  triangular  distributions  for  each  EEI.  Also,  as 
previously  mentioned,  the  current  analysis  measured  and  drew  conclusions  on  the 
efficiency  of  the  processes  being  considered.  Due  diligence  must  also  be  paid  in 
analyzing  the  effectiveness  of  the  systems  before  final  actionable  conclusions  should  be 
drawn.  Furthennore,  a  robust  analysis  of  this  study’s  true  implications  to  the  MAC 
change-over  process  is  called  for. 

On  a  larger  note,  in  terms  of  researching  SA  with  respect  to  RPA  operations, 
greater  emphasis  must  be  placed  on  the  role  of  the  CO.  While  the  pilot,  and  to  a  lesser 
extent  the  sensor  operator  roles  have  received  moderate  study  (see  Schneider  & 
McGrogan,  2011;  Chappelle  et  al,  2010;  Ouma  et  al,  2011),  little  has  been  done  with  a 
focus  on  the  CO  position  despite  its  criticality  to  complex  operations.  It  should  be 
understood  that  the  paradigm  stemming  from  manned  aircraft  of  the  pilot  having  the  most 
critical  SA  needs  does  not  typically  hold  in  modern  egocentric  medium  and  high-altitude 
RPA  operations.  At  best,  SA  needs  are  shared  equally  amongst  the  crew  and  at  worst,  the 
pilot  may  in  fact  have  a  less  substantial  SA  acquisition  and  maintenance  challenge  than 
the  communications  officer.  To  prevent  the  focus  of  future  research  from  being 
mismatched,  or  even  largely  misplaced,  a  great  deal  more  research  is  needed  to  better 
understand  the  significance  and  challenges  of  the  communications  officer  role  in  current 
and  future  RPA  operations. 
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Appendix  A.  Example  knowledge  representation  of  current  process  chronological 
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Appendix  B.  Example  knowledge  representation  of  goal-based  CTA  data 


Note  the  traceability  to  current  process  chronological  representation  via  EEI  label 
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Appendix  C.  Scholarly  Article  -  Allocation  of  Communications  to  Reduce  Mental 

Workload 

Submitted  to  Conference  on  Systems  Engineering  Research  (CSER)  2011 
Travis  Pond,  Brandon  Webster,  John  Machuca, 

John  Colombi,  Michael  Miller,  Randall  Gibb 


Abstract 

As  the  United  States  Department  of  Defense  continues  to  increase  the  number  of 
Remotely  Piloted  Aircraft  (RPA)  operations  overseas,  improved  Human  Systems 
Integration  becomes  increasingly  important.  Manpower  limitations  have  motivated  the 
investigation  of  Multiple  Aircraft  Control  (MAC)  configurations  where  a  single  pilot 
controls  multiple  RPAs  simultaneously.  Previous  research  has  indicated  that  frequent, 
unpredictable,  and  oftentimes  overwhelming,  volumes  of  communication  events  can 
produce  unmanageable  levels  of  system  induced  workload  for  MAC  pilots.  Existing 
human  computer  interface  design  includes  both  visual  infonnation  with  typed  responses, 
which  conflict  with  numerous  other  visual  tasks  the  pilot  perfonns,  and  auditory 
information  that  is  provided  through  multiple  audio  devices  with  speech  response.  This 
paper  extends  previous  discrete  event  workload  models  of  pilot  activities  flying  multiple 
aircraft.  Specifically,  we  examine  statically  reallocating  communication  modality  with 
the  goal  to  reduce  and  minimize  the  overall  pilot  cognitive  workload.  The  analysis 
investigates  the  impact  of  various  communication  reallocations  on  predicted  pilot 
workload,  measured  by  the  percent  of  time  workload  is  over  a  saturation  threshold. 
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Introduction 


Over  the  past  several  decades,  the  US  Air  Force  has  harnessed  and  exploited  the 
immense  tactical  power  that  middle  and  high-altitude  Remotely  Piloted  Aircraft  (RPAs) 
bring  to  the  battlefield.  As  a  consequence,  the  demand  for  RPA  operational  support 
continues  to  increase.  It  is  important  to  realize  that  RPAs  are  part  of  a  complex  system. 
The  system  has  many  components  including  one  or  more  air  vehicles,  ground  control 
stations  for  both  primary  mission  control  and  takeoff/landing,  a  suite  of  communications 
(including  intercom,  chat,  radios,  phones,  a  satellite  link,  etc),  support  equipment,  and 
operations  and  maintenance  crews  [1].  Assets  and  requisite  resources  to  support  those 
operations  are  limited  and  personnel  resources,  particularly  RPA  pilots,  often  prove  a 
nontrivial  constraint.  This  inevitably  leads  innovators  to  seek  out  RPA  force-multiplying 
efficiencies  to  assist  in  bridging  the  resource/demand  gap.  One  such  efficiency  being 
pursued  is  simultaneous  control  of  multiple  aircraft  by  a  single  pilot,  or  Multi  Aircraft 
Control  (MAC).  This  concept  of  operations  has  been  documented  in  the  US  Air  Force 
UAV  flight  Plan  [2]. which  calls  for  future  systems  in  which  a  single  pilot  will 
simultaneously  control  multiple  RPAs  to  enable  increased  aerial  surveillance  without 
increasing  pilot  manpower  requirements.  Previous  research  on  the  cognitive  workload 
experienced  by  pilots  during  MAC  indicated  that  frequent,  unpredictable,  and  oftentimes 
overwhelming  volumes  of  communication  events  can  produce  unmanageable  levels  of 
system  induced  workload  for  MAC  pilots  [3],  To  further  investigate  this  identified 
problem,  our  study  makes  use  of  IMPRINT  Pro,  a  Multiple  Resource  Theory  (MRT) 
based  dynamic,  stochastic  simulation  to  analyze  impacts  to  cognitive  workload  by  a 
disciplined  communication  modality  reallocation  construct. 
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Background 

In  the  RPA  domain,  communication  is  a  continuous  and  demanding  process. 
Crews  must  track,  at  a  minimum,  information  regarding  weather,  threats,  mission  tasking, 
mission  coordination,  target  coordination,  airspace  coordination,  fleet  management,  and 
status  and  location  of  any  friendly  units.  The  RPA  pilot  is  not  only  responsible  for  aircraft 
control  but  is  also  a  critical  member  in  a  multi -path  communications  infrastructure  [4],  In 
the  ground  station,  communication  with  the  pilot  takes  place  in  one  of  two  modalities: 
textual  chat  window(s)  or  the  speech-based  radio  systems.  At  any  given  moment,  a  pilot 
may  need  to  monitor  multiple  chat  windows  and  listen  to  numerous  parties  operate  over 
the  radio.  The  multitude  of  communication  sources  and  different  media  coupled  with  the 
quick  inter-arrival  rate  of  these  events  during  a  dynamic  scenario  drives  an  incredible 
cognitive  workload  for  the  pilot. 

Cognitive  or  mental  workload  expresses  the  task  demands  placed  on  an  operator 
[5],  Calculation  of  task  demand,  or  task  load,  often  considers  the  goals  of  the  operator, 
the  time  available  to  perform  the  tasks  necessary  to  accomplish  the  goals,  and  the 
performance  level  of  the  operator  [6].  Therefore,  workload  increases  when  the  number  or 
difficulty  of  tasks  necessary  to  perform  a  goal  increase,  or  when  the  times  allotted  to 
complete  these  tasks  decrease.  Assuming  that  the  operator  has  a  limited  amount  of  mental 
resources  (e.g.,  attention,  memory,  etc.)  that  he  or  she  can  utilize  to  complete  the 
necessary  tasks,  mental  workload  corresponds  to  the  proportion  of  the  operator’s  mental 
resources  demanded  by  a  task  or  set  of  tasks.  Several  methods  have  been  employed  to 
measure  and  quantify  mental  workload  over  the  past  four  decades  and  have  been 
summarized  in  numerous  publications  [5,7,8].  The  current  analysis  incorporates  Multiple 
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Resource  Theory  (MRT)  into  the  workload  calculations  to  account  for  channel  conflict 
driven  workload. 

As  a  theory,  MRT  purports  the  existence  of  four  mental  dimensions  (or  channels) 
available  to  process  information  and  perform  tasks.  The  dimensions  include  processing 
stages,  processing  codes,  perceptual  modalities  and  visual  channels.  These  channels  are 
allocated  to  concurrent  tasks  with  the  difficulty  of  the  tasks  and  the  demand  conflict 
between  channels  driving  the  overall  mental  workload  value  [9].  MRT  accurately 
describes  the  concurrent  nature  of  tasks  imposed  on  an  RPA  pilot  (performing  primary 
tasks  while  communicating  and  monitoring  communication)  and  is  therefore  an 
appropriate  theory  to  apply  to  the  present  analysis. 

Method 

Therefore,  the  specific  channels  employed  by  the  modeled  communication  events 
are  highly  relevant  to  the  MRT  workload  calculations.  As  communication  events  begin  to 
conflict  with  existing  work  activities  on  the  various  channels,  the  calculated  overall 
cognitive  workload  will  account  for  such  conflicts.  This  construct  enables  the  analysis  to 
address  the  question  of  whether  or  not  adjusting  the  intentional  allocation  of 
communication  events  to  particular  modalities  will  be  able  to  meaningfully  affect  overall 
cognitive  workload. 

Model 

A  previous  model  of  pilot  mental  workload  [3]  was  utilized  to  understand  the 
impact  of  communications  modality.  This  model  employed  functional  analysis  and  task 
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allocation  to  construct  an  executable  architecture  of  the  multiple  RPA  system.  This 
architecture  was  then  replicated  within  the  Improved  Perfonnance  Research  Integration 
Tool  (IMPRINT)  to  estimate  the  pilot’s  workload  under  various  mission  segments,  such 
as  handover,  transit,  emergency,  benign  and  dynamic  surveillance,  etc.  This  model  relied 
on  subject  matter  expert  input  to  develop  distributions  for  the  length,  frequency,  and 
difficulty  of  the  events  that  induce  workload  on  the  pilot.  The  original  research  on  this 
model  indicated  that  workload  was  particularly  high  during  what  were  termed  dynamic 
mission  segments.  These  mission  segments  often  involve  high  levels  of  communication 
between  the  pilot  and  external  actors  to  facilitate  the  tracking  or  observation  of  moving 
targets.  High  levels  of  communication  resulted  in  particularly  “high”  pilot  workload 
while  operating  a  single  aircraft  and,  “excessive”  workload  while  controlling  multiple 
dynamic-mission  aircraft.  The  original  research  indicated  that  a  reduction  in  pilot 
workload  imposed  by  communication  would  be  necessary  to  facilitate  MAC. 

To  understand  the  potential  impact  of  communication  modality  on  operator 
workload,  the  communications  portion  of  the  earlier  workload  model  was  modified  to 
permit  communications  events  to  be  reallocated  to  alternate  communications  modalities. 
The  revised  model  pennits  communication  events  that  were  originally  allocated  to  the 
auditory  channels  where  the  operator  listens  and  speaks  to  the  visual  and  fine  motor 
channels  where  the  operator  reads  and  types,  or  vice  versa. 

Figure  6  depicts  the  high  level  structure  of  the  revised  communications  model. 
The  gray  boxes  indicate  model  elements  that  were  added  to  facilitate  this  particular 
evaluation.  Communication  events  are  generated  with  a  mission  segment  dependent 
frequency  and  their  interarrival  times  are  exponentially  distributed.  In  the  original  model, 
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as  a  communication  event  is  generated,  it  is  assigned  as  either  an  auditory  event  or  a  text- 
based  event  with  25%  of  the  events  being  allocated  as  auditory  events  and  the  remaining 
allocated  as  text  events.  Half  of  the  auditory  events  then  required  the  pilot  to  talk  or  listen 
while  90%  of  the  text  events  required  the  pilot  to  read  while  only  10%  of  the  events 
required  the  pilot  to  type  a  response. 


Figure  6:  Modified  communication  model  of  pilot  workload 


To  conduct  the  current  evaluation,  the  model  was  modified  as  shown  above.  The 
auditory  and  text  events  shown  in  gray  have  the  potential  (through  a  notional  device  or 
software)  to  either  pass  an  auditory  or  text  event  as  a  respective  auditory  or  text  event  or 
to  convert  an  auditory  event  to  a  text  event  or  convert  a  text  event  to  an  auditory  event. 
With  this  modification,  it  is  assumed  that  the  characteristics  of  the  communication  are 
due  to  communication  needs,  such  that  if  a  text  event  in  the  original  model  had  a  90% 
chance  of  providing  an  input  to  the  pilot  and  only  a  10%  chance  of  an  output  to  the  pilot, 
a  text  event  converted  to  an  auditory  event  has  a  90%  probability  to  require  the  pilot  to 
listen  and  only  a  10%  probability  to  require  the  pilot  to  talk.  The  parameters  V  (for  Voice 
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reallocation)  and  T  (for  Text  reallocation)  provide  the  ability  to  convert  auditory  or  text 
events  to  its  compliment.  If  V  and  T  are  both  100%,  the  revised  model  is  the  same  as  the 
original  model.  Reducing  either  of  these  parameters  permits  a  portion  of  one  type  of 
communication  event  to  be  reallocated  to  the  complimentary  communication  event. 
Although  not  shown,  it  is  then  assumed  that  some  percentage  of  the  final  events  generate 
a  repeat  communication  event,  indicative  of  a  continued  conversation.  This  aspect  of  the 
model  was  not  changed. 

Experimental  Design 

For  this  paper,  a  total  of  six  “levels”  of  voice/text  allocation  were  selected  such 
that  the  percent  of  voice  communication  were  varied  between  0  and  100  percent.  For 
levels  of  voice  communications  less  than  25%,  V  was  varied  while  T  was  maintained  at 
100%.  However,  for  levels  of  voice  communications  greater  than  25%,  V  was  maintained 
at  100%  while  T  was  varied  to  achieve  the  desired  communications  levels.  All  analysis 
was  perfonned  for  a  10  hour  dynamic  mission  segment  with  a  single  pilot  operating  the 
aircraft.  Although  IMPRINT  does  not  currently  have  built-in  Monte  Carlo  functionality 
for  the  metrics  of  our  concern,  an  external  batch  application  was  developed  to  automate 
replications.  A  total  of  10  replications  for  each  of  six  levels  using  10  different  random 
number  seeds  were  perfonned  to  gather  the  output  data. 

The  output  of  the  IMPRINT  model  was  analyzed  to  detennine  the  proportion  of 
time  that  the  operator  would  experience  workload  values  over  a  specified  task  saturation 
threshold.  A  workload  value  of  60  was  calibrated  to  be  about  the  90%  of  operator  “red- 
line”,  which  indicates  the  workload  value  a  pilot  can  experience  without  degraded 
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performance  [10].  The  mean  and  variance  across  the  10  replications  for  each 
communication  ratio  was  calculated.  Analysis  of  Variance  (ANOVA)  and  the  Tukey 
post-hoc  tests  were  employed  detennine  the  statistical  differences  between  the  average  of 
percent  time  over  threshold. 

Results 

Figure  7  shows  the  percent  time  over  threshold  as  a  function  of  the  percentage  of 
voice  communication.  A  one  way  ANOVA  indicated  a  significant  effect  of  the  percent  of 
voice  communication  upon  the  percentage  of  time  over  threshold  (p  <  0.001).  As  shown 
in  Figure  6,  the  percent  of  time  over  threshold  is  reduced  as  the  percent  of  voice 
communication  is  increased  from  0%  to  40%.  At  40%  voice  communication  the  percent 
time  over  threshold  is  reduced  to  24.5%  compared  to  33.1%  with  0%  voice 
communication.  This  change  is  statistically  significant.  The  change  in  percent  time  over 
threshold  is  statistically  insignificant  as  the  percent  of  voice  communication  is  increased 
from  40%  to  60%.  This  trend  indicates  that  pilot  workload  is  reduced  by  the  use  of  both 
auditory  and  text-based  communications  in  this  system. 
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Comm.  Channel  Allocation 


Figure  7:  Percent  Time  Over  Threshold  as  the  percentage  of  reallocated  voice  events 


Results  further  show  that  the  percent  time  over  threshold  is  greater  at  0%  voice 
than  at  100%  voice  communications.  This  might  have  been  expected  as  reading  and 
typing  likely  conflicted  directly  with  other  tasks  being  performed  by  the  pilot,  including 
visually  monitoring  the  status  and  manipulating  the  controls  of  the  RPAs.  As  such 
workload  is  highest  when  all  of  the  communication  is  allocated  entirely  to  the  visual 
channel. 


Conclusions 

The  model  indicates  that  by  deliberately  allocating  communication  between 
auditory  and  text-based  modalities  the  pilot’s  workload  and  particularly  the  percentage  of 
time  the  pilot  operates  beyond  their  task  saturation  red-line  can  be  statistically  reduced. 
The  model  shows  that  the  percent  of  time  over  red-line  is  greatest  when  all  of  the 
communication  is  allocated  to  the  text-based  communications  such  that  zero  percent  of 
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the  communication  is  allocated  to  voice.  This  type  of  communication  is  most  likely  to 
conflict  with  other  tasks  involving  the  visual  system  to  monitor  the  RPA  and  the  small 
motor  system,  which  is  used  by  the  pilot  to  control  the  RPA.  As  communication  events 
are  moved  from  text  to  auditory,  the  workload  decreases.  However,  as  more 
communication  is  moved  to  the  auditory  channel,  the  percent  of  mission  time  over  the 
red-line  to  increases.  The  increase  likely  occurs  as  the  auditory  tasks  begin  to  overlap  and 
conflict  with  one  another  to  increase  workload.  There  appears  to  be  an  optimal  allocation 
of  communications  between  voice  and  text  modalities  to  achieve  the  lowest  workload 
given  a  constant  traffic  load.  Future  research  will  examine  dynamic  reallocation  of 
modalities. 
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